reftests: Allow minor differences to be tolerated
authorSimon McVittie <smcv@debian.org>
Sat, 13 Feb 2021 16:19:10 +0000 (16:19 +0000)
committerSimon McVittie <smcv@debian.org>
Sat, 29 Oct 2022 10:54:46 +0000 (11:54 +0100)
commit166b9ee28599662f15838548a61a4365e4c24131
treec0ac1a903f4a12d38d93424f7bf9fc8c5b259849
parent0418b0fe82786764439a84f218887ba2aa300ecc
reftests: Allow minor differences to be tolerated

Based on an earlier patch by Michael Biebl, as used in Debian's GTK 3
packaging, with additional inspiration from librsvg's reftests.

Each .ui or .node reftest can have an accompanying .keyfile file
like this:

    [reftest]
    tolerated-diff-level=20
    tolerated-diff-pixels=1000

If the image differs, but the number of pixels that differ is no more
than tolerated-diff-pixels and the differences are no more than
tolerated-diff-level, then we treat it as a success with warnings, save
the .diff.png for analysis, and use g_test_incomplete() to record the
test-case as "TODO".

Signed-off-by: Simon McVittie <smcv@debian.org>
Forwarded: https://gitlab.gnome.org/GNOME/gtk/-/merge_requests/3195
Applied-upstream: no, upstream want reftests to be a strict pass/fail with identical results required

Gbp-Pq: Name reftests-Allow-minor-differences-to-be-tolerated.patch
testsuite/gsk/compare-render.c
testsuite/reftests/gtk-reftest.c
testsuite/reftests/image-compare.c